Gestures for controlling, manipulating, and editing of media files using touch sensitive devices

ABSTRACT

Embodiments of the invention are directed to a system, method, and software for implementing gestures with touch sensitive devices (such as a touch sensitive display) for managing and editing media files on a computing device or system. Specifically, gestural inputs of a human hand over a touch/proximity sensitive device can be used to control, edit, and manipulate files, such as media files including without limitation graphical files, photo files and video files.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present invention claims the benefit under 35 USC 119(e) of U.S.provisional patent application Ser. No. 60/878,754 filed Jan. 5, 2007,the contents of which are incorporated by reference herein.

FIELD OF THE INVENTION

This relates to a system and method of managing, manipulating, andediting media objects, such as graphical objects on a display, by usinghand gestures on a touch sensitive device.

BACKGROUND OF THE INVENTION

There exist today many styles of input devices for performing operationsin a computer system. The operations generally correspond to moving acursor and making selections on a display screen. The operations canalso include paging, scrolling, panning, zooming, etc. By way ofexample, the input devices can include buttons, switches, keyboards,mice, trackballs, touch pads, joy sticks, touch screens and the like.Each of these devices has advantages and disadvantages that are takeninto account when designing a computer system.

Buttons and switches are generally mechanical in nature and providelimited control with regards to the movement of the cursor and makingselections. For example, they are generally dedicated to moving thecursor in a specific direction (e.g., arrow keys) or to making specificselections (e.g., enter, delete, number, etc.).

In using a mouse instrument, the movement of the input pointer on adisplay generally corresponds to the relative movements of the mouse asthe user moves the mouse along a surface. In using a trackballinstrument, the movement of the input pointer on the display generallycorresponds to the relative movements of a trackball as the user movesthe ball within a housing. Mouse and trackball instruments typicallyalso include one or more buttons for making selections. A mouseinstrument can also include scroll wheels that allow a user to scrollthe displayed content by rolling the wheel forward or backward.

With touch pad instrument, such as touch pads on a personal laptopcomputer, the movement of the input pointer on a display generallycorresponds to the relative movements of the user's finger (or stylus)as the finger is moved along a surface of the touch pad. Touch screens,on the other hand, can be a type of display screen that typicallyinclude a touch-sensitive transparent panel (or “skin”) that overlaysthe display screen. When using a touch screen, a user typically makes aselection on the display screen by pointing directly to objects (such asGUI objects) displayed on the screen (usually with a stylus or finger).

To provide additional functionality, hand gestures have been implementedwith some of these input devices. By way of example, in touch pads,selections may be made when one or more taps can be detected on thesurface of the touch pad. In some cases, any portion of the touch padmay be tapped, and in other cases a dedicated portion of the touch padmay be tapped. In addition to selections, scrolling may be initiated byusing finger motion at the edge of the touch pad.

U.S. Pat. Nos. 5,612,719 and 5,590,219, assigned to Apple Computer, Inc.describe some other uses of gesturing. U.S. Pat. No. 5,612,719 disclosesan onscreen button that is responsive to at least two different buttongestures made on the screen on or near the button. U.S. Pat. No.5,590,219 discloses a method for recognizing an ellipse-type gestureinput on a display screen of a computer system.

In recent times, more advanced gestures have been implemented. Forexample, scrolling may be initiated by placing four fingers on the touchpad so that the scrolling gesture is recognized and thereafter movingthese fingers on the touch pad to perform scrolling events. The methodsfor implementing these advanced gestures, however, can be limited and inmany instances counter intuitive. In certain application, especiallyapplications involving managing or editing media files using a computersystem, hand gestures using touch screens can allow a user to moreefficiently and accurately effect intended operations.

Based on the above, there is a need for improvements in the way gesturescan be performed on touch sensitive devices, especially with respect tomanaging and editing media files.

SUMMARY OF THE INVENTION

This relates to a system, method, and software for implementing gestureswith touch sensitive devices (such as a touch sensitive display) formanaging and editing media files on a computer system. Specifically,gestural inputs of a human hand over a touch/proximity sensitive devicemay be used to control, edit, and manipulate files, such as media filesincluding without limitation photo files and video files.

In accordance with one embodiment, gestural inputs over a touchsensitive computer desktop application display used to effect theconventional mouse/trackball actions, such as target, select, rightclick action, scrolling, etc.

In accordance with another embodiment, gestural inputs over a touchsensitive display may be used to effect editing commands for editingimage files, such as photo files. The gestural inputs can be recognizedvia a user interface (“UI”) element, such as a slide bar. The gesturalinputs via a UI element can be varied by changing the number oftouchdown points on the UI element.

In accordance with another embodiment, gestural inputs invoke theactivation of an UI element, after which gestural interactions with theinvoked UI element can effect further functions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer system according to an exemplaryembodiment of this invention.

FIG. 2 illustrates another computer system according to anotherexemplary embodiment of this invention.

FIG. 3 is a multipoint processing method, in accordance with anexemplary embodiment of this invention.

FIGS. 4A and 4B illustrate a detected touch image, in accordance withone embodiment of this invention.

FIG. 5 illustrates a group of features, in accordance with oneembodiment of this invention.

FIG. 6 is a parameter calculation method, in accordance with oneembodiment of this invention.

FIGS. 7A-7E and 7I-7K illustrate various gestures for performingtargeting and/or selecting tasks in accordance with one embodiment ofthis invention.

FIGS. 7F-7H show a diagram of a method for recognizing and implementinggestural inputs of FIGS. 7A-E.

FIGS. 8A-8G illustrate a rotate gesture, in accordance with oneembodiment of this invention.

FIG. 9 is a diagram of a touch-based method, in accordance with oneembodiment of this invention.

FIG. 10 is a diagram of a touch-based method, in accordance with oneembodiment of this invention.

FIG. 11 is a diagram of a touch-based method, in accordance with oneembodiment of this invention.

FIG. 12 is a diagram of a zoom gesture method, in accordance with oneembodiment of this invention.

FIGS. 13A-13H illustrates a zooming sequence, in accordance with oneembodiment of this invention.

FIG. 14 is a diagram of a pan method, in accordance with one embodimentof this invention.

FIGS. 15A-15D illustrate a panning sequence, in accordance with oneembodiment of this invention.

FIG. 16 is a diagram of a rotate method, in accordance with oneembodiment of this invention.

FIGS. 17A-17C illustrate a rotating sequence, in accordance with oneembodiment of this invention.

FIGS. 17D and 17E illustrate a method for rotating a selectable targetin accordance with one embodiment of this invention.

FIGS. 18A and 18B illustrate gestural inputs for editing a photodocument in accordance with one embodiment of this invention.

FIG. 18C is a diagram illustrating a method for recognizing andimplementing the gestural inputs of FIGS. 18A and 18B.

FIGS. 18D and 18E illustrate gestural inputs for zooming in and out of aphoto file within a photo application according to one embodiment ofthis invention.

FIGS. 19A-19D illustrate gestural inputs for scrolling through playbacksequential files according to one embodiment of this invention.

FIGS. 19E and 19F illustrate gestural inputs for scrolling throughplayback photo files on a digital camera display according to oneembodiment of this invention.

FIG. 19G illustrates gestural input for marking or deleting a photo fileduring playback according to one embodiment of this invention.

FIG. 19H illustrates an alternative gestural input for marking ordeleting a photo file during playback according to another embodiment ofthis invention.

FIG. 20 is an overview diagram showing a method for implementing themethods of FIGS. 18A-19F according to one embodiment of thisapplication.

FIGS. 21A-21D illustrate gestural inputs for controlling and/or editingvideo using a video application according to one embodiment of thisinvention.

FIGS. 22A and 22B are diagrams of a method for implementing the gesturalinputs of FIGS. 21A-21D.

FIG. 23 illustrate gestural inputs for controlling and/or editing audiousing an audio application according to one embodiment of thisinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description of preferred embodiments, reference is madeto the accompanying drawings which form a part hereof, and in which itis shown by way of illustration specific embodiments in which thisinvention can be practiced. It is to be understood that otherembodiments can be utilized and structural changes can be made withoutdeparting from the scope of the preferred embodiments of the invention.

FIG. 1 is a block diagram of an exemplary computer system 50, in.accordance with one embodiment of the invention. The computer system 50can correspond to a personal computer system, such as a desktops,laptops, tablets or handheld computer. The computer system can alsocorrespond to a computing device, such as a cell phone, PDA, dedicatedmedia player, consumer electronic device, and the like.

The exemplary computer system 50 shown in FIG. 1 can include a processor56 configured to execute instructions and to carry out operationsassociated with the computer system 50. For example, using instructionsretrieved for example from memory, the processor 56 can control thereception and manipulation of input and output data between componentsof the computing system 50. The processor 56 can be implemented on asingle-chip, multiple chips or multiple electrical components. Forexample, various architectures can be used for the processor 56,including dedicated or embedded processor, single purpose processor,controller, ASIC, and so forth.

In most cases, the processor 56 together with an operating systemoperates to execute computer code and produce and use data. Operatingsystems are generally well known and will not be described in greaterdetail. By way of example, the operating system can correspond to OS/2,DOS, Unix, Linux, Palm OS, and the like. The operating system can alsobe a special purpose operating system, such as ones that can be used forlimited purpose appliance-type computing devices. The operating system,other computer code and data can reside within a memory block 58 thatcan be operatively coupled to the processor 56. Memory block 58generally provides a place to store computer code and data that can beused by the computer system 50. By way of example, the memory block 58can include Read-Only Memory (ROM), Random-Access Memory (RAM), harddisk drive and/or the like. The information could also reside on aremovable storage medium and loaded or installed onto the computersystem 50 when needed. Removable storage mediums include, for example,CD-ROM, PC-CARD, memory card, floppy disk, magnetic tape, and a networkcomponent.

The computer system 50 can also include a display device 68 that can beoperatively coupled to the processor 56. The display device 68 can be aliquid crystal display (LCD) (e.g., active matrix, passive matrix andthe like). Alternatively, the display device 68 can be a monitor such asa monochrome display, color graphics adapter (CGA) display, enhancedgraphics adapter (EGA) display, variable-graphics-array (VGA) display,super VGA display, cathode ray tube (CRT), and the like. The displaydevice can also correspond to a plasma display or a display implementedwith electronic inks.

The display device 68 can be generally configured to display a graphicaluser interface (GUI) 69 that provides an easy to use interface between auser of the computer system and the operating system or applicationrunning thereon. Generally speaking, the GUI 69 represents, programs,files and operational options with graphical images, objects, or vectorrepresentations. The graphical images can include windows, fields,dialog boxes, menus, icons, buttons, cursors, scroll bars, etc. Suchimages may be arranged in predefined layouts, or can be createddynamically to serve the specific actions being taken by a user. Duringoperation, the user can select and/or activate various graphical imagesin order to initiate functions and tasks associated therewith. By way ofexample, a user may select a button that opens, closes, minimizes, ormaximizes a window, or an icon that launches a particular program. TheGUI 69 can additionally or alternatively display information, such asnon interactive text and graphics, for the user on the display device68.

The computer system 50 can also include an input device 70 that can beoperatively coupled to the processor 56. The input device 70 can beconfigured to transfer data from the outside world into the computersystem 50. The input device 70 may for example be used to performtracking and to make selections with respect to the GUI 69 on thedisplay 68. The input device 70 may also be used to issue commands inthe computer system 50. The input device 70 can include a touch sensingdevice configured to receive input from a user's touch and to send thisinformation to the processor 56. By way of example, the touch-sensingdevice can correspond to a touchpad or a touch screen. In many cases,the touch-sensing device recognizes touches, as well as the position andmagnitude of touches on a touch sensitive surface. The touch sensingdevice detects and reports the touches to the processor 56 and theprocessor 56 interprets the touches in accordance with its programming.For example, the processor 56 can initiate a task in accordance with aparticular touch. A dedicated processor can be used to process toucheslocally and reduce demand for the main processor of the computer system.

The touch sensing device can be based on sensing technologies includingbut not limited to capacitive sensing, resistive sensing, surfaceacoustic wave sensing, pressure sensing, optical sensing, and/or thelike. Furthermore, the touch sensing means can be based on single pointsensing or multipoint sensing. Single point sensing is capable of onlydistinguishing a single touch, while multipoint sensing can be capableof distinguishing multiple touches that occur at the same time.

As discussed above, the input device 70 can be a touch screen that ispositioned over or in front of the display 68, integrated with thedisplay device 68, or can be a separate component, such as a touch pad.

The computer system 50 also preferably includes capabilities forcoupling to one or more I/O devices 80. By way of example, the I/Odevices 80 can correspond to keyboards, printers, scanners, cameras,microphones, speakers, and/or the like. The I/O devices 80 can beintegrated with the computer system 50 or they can be separatecomponents (e.g., peripheral devices). In some cases, the I/O devices 80can be connected to the computer system 50 through wired connections(e.g., cables/ports). In other cases, the I/O devices 80 can beconnected to the computer system 80 through wireless connections. By wayof example, the data link can correspond to PS/2, USB, IR, Firewire, RF,Bluetooth or the like.

In accordance with one embodiment of the invention, the computer system50 is designed to recognize gestures 85 applied to the input device 70and to control aspects of the computer system 50 based on the gestures85. In some cases, a gesture can be defined as a stylized interactionwith an input device that can be mapped to one or more specificcomputing operations. The gestures 85 can be made through various hand,and more particularly finger motions. Alternatively or additionally, thegestures can be made with a stylus. In all of these cases, the inputdevice 70 receives the gestures 85 and the processor 56 executesinstructions to carry out operations associated with the gestures 85. Inaddition, the memory block 58 can include a gesture operational program88, which can be part of the operating system or a separate application.The gestural operation program 88 generally can include a set ofinstructions that recognizes the occurrence of gestures 85 and informsone or more software agents of the gestures 85 and/or what action(s) totake in response to the gestures 85. Additional details regarding thevarious gestures that can be used as input commands is discussed furtherbelow.

In accordance with a preferred embodiment, upon a user performing one ormore gestures, the input device 70 relays gesture information to theprocessor 56. Using instructions from memory 58, and more particularly,the gestural operational program 88, the processor 56 interprets thegestures 85 and controls different components of the computer system 50,such as memory 58, a display 68 and I/O devices 80, based on thegestures 85. The gestures 85 may be identified as commands forperforming actions in applications stored in the memory 58, modifyingimage objects shown on the display 68, modifying data stored in memory58, and/or for performing actions in I/O devices 80.

Again, although FIG. 1 illustrates the input device 70 and the display68 as two separate boxes for illustration purposes, the two boxes can berealized on one device.

FIG. 2 illustrates an exemplary computing system 10 that uses amulti-touch panel 24 as an input device for gestures; the multi-touchpanel 24 can at the same time be a display panel. The computing system10 can include one or more multi-touch panel processors 12 dedicated tothe multi-touch subsystem 27. Alternatively, the multi-touch panelprocessor functionality can be implemented by dedicated logic, such as astate machine. Peripherals 11 can include, but are not limited to,random access memory (RAM) or other types of memory or storage, watchdogtimers and the like. Multi-touch subsystem 27 can include, but is notlimited to, one or more analog channels 17, channel scan logic 18 anddriver logic 19. Channel scan logic 18 can access RAM 16, autonomouslyread data from the analog channels and provide control for the analogchannels. This control can include multiplexing columns of multi-touchpanel 24 to analog channels 17. In addition, channel scan logic 18 cancontrol the driver logic and stimulation signals being selectivelyapplied to rows of multi-touch panel 24. In some embodiments,multi-touch subsystem 27, multi-touch panel processor 12 and peripherals11 can be integrated into a single application specific integratedcircuit (ASIC).

Driver logic 19 can provide multiple multi-touch subsystem outputs 20and can present a proprietary interface that drives high voltage driver,which preferably includes a decoder 21 and subsequent level shifter anddriver stage 22, although level-shifting functions could be performedbefore decoder functions. Level shifter and driver 22 can provide levelshifting from a low voltage level (e.g. CMOS levels) to a higher voltagelevel, providing a better signal-to-noise (S/N) ratio for noisereduction purposes. Decoder 21 can decode the drive interface signals toone out of N outputs, whereas N can be the maximum number of rows in thepanel. Decoder 21 can be used to reduce the number of drive lines neededbetween the high voltage driver and multi-touch panel 24. Eachmulti-touch panel row input 23 can drive one or more rows in multi-touchpanel 24. It should be noted that driver 22 and decoder 21 can also beintegrated into a single ASIC, be integrated into driver logic 19, or insome instances be unnecessary.

The multi-touch panel 24 can include a capacitive sensing medium havinga plurality of row traces or driving lines and a plurality of columntraces or sensing lines, although other sensing media can also be used.The row and column traces can be formed from a transparent conductivemedium, such as Indium Tin Oxide (ITO) or Antimony Tin Oxide (ATO),although other transparent and non-transparent materials, such ascopper, can also be used. In some embodiments, the row and column tracescan be formed on opposite sides of a dielectric material, and can beperpendicular to each other, although in other embodiments othernon-Cartesian orientations are possible. For example, in a polarcoordinate system, the sensing lines can be concentric circles and thedriving lines can be radially extending lines (or vice versa). It shouldbe understood, therefore, that the terms “row” and “column,” “firstdimension” and “second dimension,” or “first axis” and “second axis” asused herein are intended to encompass not only orthogonal grids, but theintersecting traces of other geometric configurations having first andsecond dimensions (e.g. the concentric and radial lines of apolar-coordinate arrangement). The rows and columns can be formed on asingle side of a substrate, or can be formed on two separate substratesseparated by a dielectric material. In some instances, an additionaldielectric cover layer can be placed over the row or column traces tostrengthen the structure and protect the entire assembly from damage.

At the “intersections” of the traces of the multi-touch panel 24, wherethe traces pass above and below (cross) each other (but do not makedirect electrical contact with each other), the traces essentially formtwo electrodes (although more than two traces could intersect as well).Each intersection of row and column traces can represent a capacitivesensing node and can be viewed as picture element (pixel) 26, which canbe particularly useful when multi-touch panel 24 is viewed as capturingan “image” of touch. (In other words, after multi-touch subsystem 27 hasdetermined whether a touch event has been detected at each touch sensorin the multi-touch panel, the pattern of touch sensors in themulti-touch panel at which a touch event occurred can be viewed as an“image” of touch (e.g. a pattern of fingers touching the panel).) Thecapacitance between row and column electrodes appears as a straycapacitance on all columns when the given row is held at DC and as amutual capacitance Csig when the given row is stimulated with an ACsignal. The presence of a finger or other object near or on themulti-touch panel can be detected by measuring changes to Csig. Thecolumns of multi-touch panel 124 can drive one or more analog channels17 (also referred to herein as event detection and demodulationcircuits) in multi-touch subsystem 27. In some implementations, eachcolumn can be coupled to one dedicated analog channel 17. However, inother implementations, the columns can be couplable via an analog switchto a fewer number of analog channels 17.

Computing system 10 can also include host processor 14 for receivingoutputs from multi-touch panel processor 12 and performing actions basedon the outputs that can include, but are not limited to, moving anobject such as a cursor or pointer, scrolling or panning, adjustingcontrol settings, opening a file or document, viewing a menu, making aselection, executing instructions, operating a peripheral deviceconnected to the host device, etc. Host processor 14, which can be apersonal computer CPU, can also perform additional functions that maynot be related to multi-touch panel processing, and can be coupled toprogram storage 15 and display device 13 such as an LCD display forproviding a user interface (UI) to a user of the device.

It should be noted that, while FIG. 2 illustrates a dedicated MT panelprocessor 12, the multi-touch subsystem may be controlled directly bythe host processor 14. Additionally, it should also be noted that themulti-touch panel 24 and the display device 13 can be integrated intoone single touch-screen display device. Further details of multi-touchsensor detection, including proximity detection by a touch panel, aredescribed in commonly assigned co-pending applications, includingapplication Ser. No. 10/840,862, published as U.S. patent publicationno. US2006/0097991, application Ser. No. 11/428,522, published as U.S.patent publication no. US2006/0238522, and application titled “Proximityand Multi-Touch Sensor Detection and Demodulation,” filed on Jan. 3,2007, the entirety of all of which are hereby incorporated herein byreference.

FIG. 3 illustrates a multipoint processing method 100, in accordancewith one embodiment of the invention. The multipoint processing method100 may for example be performed in the system shown in FIGS. 1 or 2.The multipoint processing method 100 generally begins at block 102 whereimages can be read from a multipoint input device, and more particularlya multipoint touch screen. Although the term “image” is used it shouldbe noted that the data may come in other forms. In most cases, the imageread from the touch screen provides magnitude (Z) as a function ofposition (x and y) for each sensing point or pixel of the touch screen.The magnitude may, for example, reflect the capacitance measured at eachpoint.

Following block 102, multipoint processing method 100 proceeds to block104 where the image can be converted into a collection or list offeatures. Each feature represents a distinct input such as a touch. Inmost cases, each feature can include its own unique identifier (ID), xcoordinate, y coordinate, Z magnitude, angle Θ, area A, and the like. Byway of example, FIGS. 4A and 4B illustrate a particular image 120 intime. In image 120, there can be two features 122 based on two distincttouches. The touches may for example be formed from a pair of fingerstouching the touch screen. As shown, each feature 122 can include uniqueidentifier (ID), x coordinate, y coordinate, Z magnitude, angle .theta.,and area A. More particularly, the first feature 122A can be representedby ID₁, X₁, Y₁, Z₁, Θ₁, A₁and the second feature 122B can be representedby ID₂, X₂, Y₂, Z₂, Θ₂, A₂. This data may be outputted for example usinga multi-touch protocol.

The conversion from data or images to features may be accomplished usingmethods described in copending U.S. patent application Ser. No.10/840,862, published as U.S. patent publication no. US2006/007991,which is hereby again incorporated herein by reference. As disclosedtherein, the raw data is typically received in a digitized form, and caninclude values for each node of the touch screen; The values can bebetween 0 and 256 where 0 equates to no touch pressure and 256 equate tofull touch pressure. Thereafter, the raw data can be filtered to reducenoise. Once filtered, gradient data, which indicates the topology ofeach group of connected points, can be generated. Thereafter, theboundaries for touch regions can be calculated based on the gradientdata (i.e., a determination can be made as to which points can begrouped together to form each touch region). By way of example, awatershed algorithm may be used. Once the boundaries are determined, thedata for each of the touch regions can be calculated (e.g., X, Y, Z, θ,A).

Following block 104, multipoint processing method 100 proceeds to block106 where feature classification and groupings can be performed. Duringclassification, the identity of each of the features can be determined.For example, the features may be classified as a particular finger,thumb, palm or other object. Once classified, the features may begrouped. The manner in which the groups are formed can widely vary. Inmost cases, the features can be grouped based on some criteria (e.g.,they carry a similar attribute). For example, the two features shown inFIGS. 4A and 4B may be grouped together because each of these featurescan be located in proximity to each other or because they are from thesame hand. The grouping may include some level of filtering to filterout features that are not part of the touch event. In filtering, one ormore features may be rejected because they either meet some predefinedcriteria or because they do not meet some criteria. By way of example,one of the features may be classified as a thumb located at the edge ofa tablet PC. Because the thumb is being used to hold the device ratherthan being used to perform a task, the feature generated therefrom isrejected, i.e., is not considered part of the touch event beingprocessed.

Following block 106, the multipoint processing method 100 proceeds toblock 108 where key parameters for the feature groups can be calculated.The key parameters may include distance between features, x/y centroidof all features, feature rotation, total pressure of the group (e.g.,pressure at centroid), and the like. As shown in FIG. 5, the calculationmay include finding the centroid C, drawing a virtual line 130 to eachfeature from the centroid C, defining the distance D for each virtualline (D₁ and D₂), and then averaging the distances D₁ and D₂. Once theparameters are calculated, the parameter values can be reported. Theparameter values are typically reported with a group identifier (GID)and number of features within each group (in this case three). In mostcases, both initial and current parameter values are reported. Theinitial parameter values may be based on set down, i.e., when the usersets their fingers on the touch screen, and the current values may bebased on any point within a stroke occurring after set down.

As should be appreciated, blocks 102-108 can be repetitively performedduring a user stroke thereby generating a plurality of sequentiallyconfigured signals. The initial and current parameters can be comparedin later steps to perform actions in the system.

Following block 108, the process flow proceeds to block 110 where thegroup is or can be associated with a user interface (UI) element. UIelements can be buttons boxes, lists, sliders, wheels, knobs, etc. EachUI element represents a component or control of the user interface. Theapplication behind the UI element(s) can have access to the parameterdata calculated in block 108. In one implementation, the applicationranks the relevance of the touch data to the UI element correspondingthere to. The ranking may be based on some predetermine criteria. Theranking may include producing a figure of merit, and whichever UIelement has the highest figure of merit, giving it sole access to thegroup. There may even be some degree of historesis as well (once one ofthe UI elements claims control of that group, the group sticks with theUI element until another UI element has a much higher ranking). By wayof example, the ranking may include determining proximity of thecentroid (or features) to the image object associated with the UIelement.

Following block 110, the multipoint processing method 100 proceeds toblocks 112 and 114. The blocks 112 and 114 can be performedapproximately at the same time. From the user perspective, in oneembodiment, the blocks 112 and 114 appear to be performed concurrently.In block 112, one or more actions can be performed based on differencesbetween initial and current parameter values, and may also be based to aUI element to which they are associated, if any. In block 114, userfeedback pertaining to the one ore more action being performed can beprovided. By way of example, user feedback may include display, audio,tactile feedback and/or the like.

FIG. 6 is a parameter calculation method 150, in accordance with oneembodiment of the invention. The parameter calculation method 150 can,for example, correspond to block 108 shown in FIG. 3. The parametercalculation method 150 generally begins at block 152 where a group offeatures can be received. Following block 152, the parameter calculationmethod 150 proceeds to block 154 where a determination can be made as towhether or not the number of features in the group of features haschanged. For example, the number of features may have changed due to theuser picking up or placing an additional finger. Different fingers maybe needed to perform different controls (e.g., tracking, gesturing). Ifthe number of features has changed, the parameter calculation method 150proceeds to block 156 where the initial parameter values can becalculated. If the number stays the same, the parameter calculationmethod 150 proceeds to block 158 where the current parameter values canbe calculated. Thereafter, the parameter calculation method 150 proceedsto block 150 where the initial and current parameter values can bereported. By way of example, the initial parameter values may containthe average initial distance between points (or Distance (AVG) initial)and the current parameter values may contain the average currentdistance between points (or Distance (AVG) current). These may becompared in subsequent steps in order to control various aspects of acomputer system.

The above methods and techniques can be used to implement any number ofGUI interface objects and actions. For example, gestures can be createdto detect and effect a user command to resize a window, scroll adisplay, rotate an object, zoom in or out of a displayed view, delete orinsert text or other objects, etc.

A basic category of gestures should allow a user to input the commoncommands that can be inputted through the use of a conventional mouse ortrackball instrument. FIG. 7F shows a flow chart for processing thedetection of mouse-click actions. Starting with block 710, detection canbe made of either one or two touches by fingers. If the touch detectedcan be determined 711 to be one finger, then a determination 712 can bemade of whether the touch is in a predetermined proximity of a displayedimage object that is associated with a selectable file object, and ifso, then a selection action is made 714. If a double tap action isdetected 716 in association with a selectable object, then adouble-click action can be invoked 718. A double tap action can bedetermined by the detection of a finger leaving the touch screen andimmediately retouching the touch screen twice. In accordance with analternative embodiment, a double-click action can also be invoked if itis detected that a finger touch on a selected object remains for morethan a predetermined period of time.

As shown in FIG. 7G, if the one finger touch detected is not associatedwith a selectable file object, but rather is determined 720 to beassociated with a network address hyperlink, then a single-click actioncan be invoked whereby the hyperlink can be activated. If the hyperlinkwas touched within a non-browser environment, then a browser applicationwould also be launched.

If a two finger touch is detected 711, then if at least one of thetouchdown points is associated with a selectable file object 713, then aselection 715 is made of the object. If one or more tap by one of thefingers on the touch sensitive display is detected 717 while thetouchdown point is maintained, then a right-click mouse action can beinvoked.

In accordance with a preferred embodiment, if a touch or touchesdetected are not associated with any selectable file object orhyperlinks, then as shown in FIG. 7H, a determination 722 can be made asto whether the touchdown point(s) is/can be associated with a scrollablearea, such as text editing application window, a file listing window, oran Internet webpage.

Scrolling generally pertains to moving displayed data or images across aviewing area on a display screen so that a new set of data can bebrought into view in the viewing area. In most cases, once the viewingarea is full, each new set of data appears at the edge of the viewingarea and all other sets of data move over one position. That is, the newset of data appears for each set of data that moves out of the viewingarea. In essence, these functions allow a user to view consecutive setsof data currently outside of the viewing area. In most cases, the useris able to accelerate their traversal through the data sets by movinghis or her finger at greater speeds. Examples of scrolling through listscan be found in U.S. Patent Publication Nos. 2003/0076303A1,2003/0076301A1, 2003/0095096A1, which are herein incorporated byreference.

If the touch down point(s) is/can be within a scrollable area, then ascrolling action can be invoked 723 similar to the pressing down of ascroll wheel on a conventional mouse instrument. If the scrollable areais scrollable in only one direction (e.g., up and down), then thescrolling action invoked will be unidirectional scroll. If thescrollable area is scrollable two dimensionally, then the scrollingaction invoked will be omnidirectional.

In a unidirectional scrolling action where the scrolling can berestricted to the vertical direction (i.e., the Y axis), the only thevertical vector component of the tracked touch movement will be used asinput for effecting vertical scrolling. Similarly, in a unidirectionalscrolling action where the scrolling is restricted to the horizontaldirection (i.e., the X axis), the only the horizontal vector componentof the tracked touch movement will be used as input for effectinghorizontal scrolling. If the scrolling action is omnidirectional, thenthe scrolling action effected will track the movement of the trackedtouch.

In accordance with a preferred embodiment, if the detected touch is aone finger touch, then the scrolling action can be ready to be performed724 at a normal, or 1X, speed. If and once the touched down fingerbegins to move on the touch screen, then a scroll action can beperformed by tracking the movement of the touchdown point on the touchscreen. If the detected touch is a two finger touch, then the scrollingaction can be performed 725 at a double, or 2X speed. Additional fingersmay still be added to perform even faster scrolling action, where adetection of a four finger touch may be translated into “pg up” or “pgdn” commands within a multi-page document window.

In accordance with another embodiment, the displayed data continues tomove even when the finger is removed from the touch screen. Thecontinuous motion can be based at least in part on the previous motion.For example the scrolling can be continued in the same direction andspeed. In some cases, the scrolling slow down over time, i.e., the speedof the traversal through the media items gets slower and slower untilthe scrolling eventually stops thereby leaving a static list. By way ofexample, each new media item brought into the viewing area mayincrementally decrease the speed. Alternatively or additionally, thedisplayed data stops moving when the finger is placed back on the touchscreen. That is, the placement of the finger back on the touch screencan implement braking, which stops or slows down the continuous actingmotion.

By way of examples to illustrate the above discussed gestural actions,as shown in FIG. 7A, using a touch screen (such as the multi-touchscreen 24 shown in FIG. 2), a single finger tap with a finger 501 on animage object (e.g., a file listing 500) may be translated into theequivalent of a single click of a mouse, which in this instance mayindicate a selection, which is typically indicated by highlighting ofthe selected file or image object. A detected double tap on the imageobject may be translated into the equivalent of a double click of amouse, which may invoke a launch of an application associated with theimage object tapped. For instance, a double tapping of a file listing ona screen, such as a photo file, may cause the launch of a photo viewerapplication and the opening of that photo file.

Drag-and-drop function can be invoked by touching, within at least onefinger, the image associated with the object to be dropped andgraphically dragging the object to the desired drop location bymaintaining the touch, such as shown in FIG. 7B, illustrating a drag anddrop of a file listing 501 from folder window 502 to folder window 503.

Certain mouse functionalities may require two touches to complete. Forinstance, as shown in FIG. 7C, a “right click” gesture can be invoked bytwo fingers, with one finger as the touchdown finger 506 and a secondfinger 507 tapping the screen at least once to indicate a right clickaction. FIG. 7D illustrates that, after a right click action can beperformed, an action window 504 can be invoked, after which the firstfinger can move over to the invoked window 504 to select and tap anaction item 505 with a single finger 506. In accordance with oneembodiment of this invention, a right click action can be effected onlyif the tapping detected is located in close proximity of the detectedtouchdown, and only if the tapping detected is located to the left ofthe touchdown finger (right of the touchdown finger from the user'spoint of view).

Other file selection functions that normally require a combination ofmouse and keyboard action can be performed using only touch action. Forinstance, in the Microsoft Windows environment, to select multiple fileswithin file window 502, a user typically needs to hold down the shiftbutton while dragging the mouse icon over the sequential files to beselected. Without holding down the shift button, the dragging of themouse icon may be interpreted as a drag and drop action. As shown inFIG. 7E, in accordance with an embodiment of the invention, a detectionof two closely associated touch drag of file listings may be read as amulti-selection action for selecting a group of files 508. In order toavoid misinterpreting the two-touch action as another command, such as arotating action, the two-touch multi-selection function is preferablyinvoked only if the two touches detected are in relative close proximityto each other.

Referring to the scrolling actions described in FIG. 7H, and as shown inFIGS. 7I and 7J, a one or two finger touchdown within a scrollablewindow may cause the displayed content of the window to scroll atdifferent speeds. Specifically, once a scrolling action is invoked 723,the scrolling takes place a 1X speed 724 if it is determined that onlyone finger (or one touchdown point) is detected on the touch sensitivedisplay, and at 2X speed if two fingers (or two touchdown points) aredetected. In accordance with a preferred embodiment, during the scrollaction, scroll bars 727 and 728 move in correspondence to the directionof the scrolls.

Finally, using a multi-touch display that is capable of proximitydetection, such as the panels described in the aforementioned andincorporated by reference commonly assigned co-pending applications Ser.No. 10/840,862 (published on as U.S. patent publication no.US2006/0097991) and application titled “Proximity and Multi-Touch SensorDetection and Demodulation,” filed on Jan. 3, 2007, gestures of a fingercan also be used to invoke hovering action that can be the equivalent ofhovering a mouse icon over an image object.

By way of an example, referring to FIG. 7K, the detection of proximityof a user's finger 501 over application icons 731 within a desktop 729can be interpreted as a hovering action, which invokes the rolling popupof the hovered application icon 730. If the user touches the popped upicon, then a double-click action can be invoked whereby the applicationcan be launched. Similar concepts can be applied to application specificsituations, such as when photo files are displayed in a thumbnail formatwithin a photo management software, and a detection of proximity of afinger over a thumbnail invokes a hover action whereby the size of thehovered photo thumbnail can be enlarged (but not selected).

Gestures can also be used to invoke and manipulate virtual controlinterfaces, such as volume knobs, switches, sliders, keyboards, andother virtual interfaces that can be created to facilitate humaninteraction with a computing system or a consumer electronic item. Byway of an example using a gesture to invoke a virtual control interface,and referring to FIGS. 8A-8H, a rotate gesture for controlling a virtualvolume knob 170 on a GUI interface 172 of a display 174 of a tablet PC175 will be described. In order to actuate the knob 170, the user placestheir fingers 176 on a multipoint touch screen 178. The virtual controlknob may already be displayed, or the particular number, orientation orprofile of the fingers at set down, or the movement of the fingersimmediately thereafter, or some combination of these and othercharacteristics of the user's interaction may invoke the virtual controlknob to be displayed. In either case, the computing system associates afinger group to the virtual control knob and makes a determination thatthe user intends to use the virtual volume knob.

This association may also be based in part on the mode or current stateof the computing device at the time of the input. For example, the samegesture can be interpreted alternatively as a volume knob gesture if asong is currently playing on the computing device, or as a rotatecommand if an object editing application is being executed. Other userfeedback can be provided, including for example audible or tactilefeedback.

Once the knob 170 is displayed as shown in FIG. 8A, the user's fingers176 can be positioned around the knob 170 similar to if it were anactual knob or dial, and thereafter can be rotated around the knob 170in order to simulate turning the knob 170. Again, audible feedback inthe form of a clicking sound or tactile feedback in the form ofvibration, for example, can be provided as the knob 170 can be“rotated.” The user may also use his or her other hand to hold thetablet PC 175.

As shown in FIG. 8B, the multipoint touch screen 178 detects at least apair of images. In particular, a first image 180 is created at set down,and at least one other image 182 can be created when the fingers 176 arerotated. Although only two images are shown, in most cases there wouldbe many more images that incrementally occur between these two images.Each image represents a profile of the fingers in contact with the touchscreen at a particular instant in time. These images can also bereferred to as touch images. It will be understood that the term “image”does not mean that the profile is displayed on the screen 178 (butrather imaged by the touch sensing device). It should also be noted thatalthough the term “image” is used, the data can be in other formsrepresentative of the touch plane at various times.

As shown in FIG. 8C, each of the images 180 and 182 can be converted toa collection of features 184. Each feature 184 can be associated with aparticular touch as for example from the tips each of the fingers 176surrounding the knob 170 as well as the thumb of the other hand 177 usedto hold the tablet PC 175.

As shown in FIG. 8D, the features 184 are classified, i.e., eachfinger/thumb is identified, and grouped for each of the images 180 and182. In this particular case, the features 184A associated with the knob170 can be grouped together to form group 188 and the feature 184Bassociated with the thumb can be filtered out. In alternativearrangements, the thumb feature 184B may be treated as a separatefeature by itself (or in another group), for example, to alter the inputor operational mode of the system or to implement another gesture, forexample, a slider gesture associated with an equalizer slider displayedon the screen in the area of the thumb (or other finger).

As shown in FIG. 8E, the key parameters of the feature group 188 can becalculated for each image 180 and 182. The key parameters associatedwith the first image 180 represent the initial state and the keyparameters of the second image 182 represent the current state.

Also as shown in FIG. 8E, the knob 170 is the UI element associated withthe feature group 188 because of its proximity to the knob 170.Thereafter, as shown in FIG. 8F, the key parameter values of the featuregroup 188 from each image 180 and 182 can be compared to determine therotation vector, i.e., the group of features rotated five (5) degreesclockwise from the initial to current state. In FIG. 8F, the initialfeature group (image 180) is shown in dashed lines while the currentfeature group (image 182) is shown in solid lines.

As shown in FIG. 8G, based on the rotation vector the speaker 192 of thetablet PC 175 increases (or decreases) its output in accordance with theamount of rotation of the fingers 176, i.e., increase the volume by 5%based on rotation of 5 degrees. The display 174 of the tablet PC canalso adjust the rotation of the knob 170 in accordance with the amountof rotation of the fingers 176, i.e., the position of the knob 170rotates five (5) degrees. In most cases, the rotation of the knob occurssimultaneously with the rotation of the fingers, i.e., for every degreeof finger rotation the knob rotates a degree. In essence, the virtualcontrol knob follows the gesture occurring on the screen. Still further,an audio unit 194 of the tablet PC can provide a clicking sound for eachunit of rotation, e.g., provide five clicks based on rotation of fivedegrees. Still yet further, a haptics unit 196 of the tablet PC 175 canprovide a certain amount of vibration or other tactile feedback for eachclick thereby simulating an actual knob.

It should be noted that additional gestures can be performedsimultaneously with the virtual control knob gesture. For example, morethan one virtual control knob can be controlled at the same time usingboth hands, i.e., one hand for each virtual control knob. Alternativelyor additionally, one or more slider bars can be controlled at the sametime as the virtual control knob, i.e., one hand operates the virtualcontrol knob, while at least one finger and alternatively more than onefinger of the opposite hand operates at least one slider andalternatively more than one slider bar, e.g., slider bar for eachfinger.

It should also be noted that although the embodiment is described usinga virtual control knob, in another embodiment, the UI element can be avirtual scroll wheel. As an example, the virtual scroll wheel can mimican actual scroll wheel such as those described in U.S. patentpublication nos. US2003/0076303A1, US2003/0076301A1, andUS2003/0095096A1, all of which are herein incorporated by reference.

FIG. 9 is a diagram of a touch-based method 200 in accordance with oneembodiment of the invention. The method generally begins at block 202where a user input that occurs over a multipoint sensing device can bedetected. The user input can include one or more touch inputs, with eachtouch input having a unique identifier. Following block 202, thetouch-based method 200 proceeds to block 204 where the user input can beclassified as a tracking or selection input when the user input caninclude a single unique identifier (one touch input), or can beclassified as a gesture input when the user input can include at leasttwo unique identifiers (more than one touch input). If the user inputcan be classified as a tracking input, the touch-based method 200proceeds to block 206 where tracking can be performed corresponding tothe user input.

If the user input is classified as a gesture input, the touch-basedmethod 200 proceeds to block 208 where one or more gesture controlactions can be performed corresponding the user input. The gesturecontrol actions can be based at least in part on changes that occur withor between the at least two unique identifiers.

FIG. 10 is a diagram of a touch-based method 250 in accordance with oneembodiment of the invention. The touch-based method 250 generally beginsat block 252 where an initial image can be captured during an inputstroke on a touch sensitive surface. Following block 252, thetouch-based method 250 proceeds to block 254 where the touch mode can bedetermined based on the initial image. For example, if the initial imageincludes a single unique identifier then the touch mode may correspondto a tracking or selection mode. On the other hand, if the imageincludes more than one unique identifier, then the touch mode maycorrespond to a gesture mode.

Following block 254, the touch-based method 250 proceeds to block 256where a next image can be captured during the input stroke on the touchsensitive surface. Images can be typically captured sequentially duringthe stroke and thus the there may be a plurality of images associatedwith the stroke.

Following block 256, touch-based method 250 proceeds to block 258 wherea determination can be made as to whether the touch mode changed betweencapture of the initial image and capture of the next image. If the touchmode changed, the touch-based method 250 proceeds to block 260 where thenext image can be set as the initial image and thereafter the touch modeis again determined at block 254 based on the new initial image. If thetouch mode stayed the same, the touch-based method 250 proceeds to block262 where the initial and next images can be compared and one or morecontrol signals can be generated based on the comparison.

FIG. 11 is a diagram of a touch-based method 300 in accordance with oneembodiment of the invention. The touch-based method 300 begins at block302 where an image object, which can be a GUI object, can be output. Forexample, a processor may instruct a display to display a particularimage object. Following block 302, the touch-based method 300 proceedsto block 304 where a gesture input is received over the image object.For instance, a user may set or move their fingers in a gestural way onthe surface of the touch screen and while over the displayed imageobject. The gestural input may include one or more single gestures thatoccur consecutively or multiple gestures that occur simultaneously. Eachof the gestures generally has a particular sequence, motion, ororientation associated therewith. For example, a gesture may includespreading fingers apart or closing fingers together, rotating thefingers, translating the fingers, and/or the like.

Following block 304 the touch-based method 300 proceeds to block 306where the Image object can be modified based on and in unison with thegesture input. By modified, it is meant that the image object changesaccording to the particular gesture or gestures being performed. By inunison, it is meant that the changes occur approximately while thegesture or gestures are being performed. In most cases, there is a oneto one relationship between the gesture(s) and the changes occurring atthe image object and they occur substantially simultaneously. Inessence, the image object follows the motion of the fingers. Forexample, spreading of the fingers may simultaneously enlarge the object,closing of the fingers may simultaneously reduce the image object,rotating the fingers may simultaneously rotate the object, translatingthe fingers may allow simultaneous panning or scrolling of the imageobject.

In one embodiment, block 306 can include determining which image objectis associated with the gesture being performed, and thereafter lockingthe displayed object to the fingers disposed over it such that the imageobject changes in accordance with the gestural input. By locking orassociating the fingers to the image object, the image object cancontinuously adjust itself in accordance to what the fingers are doingon the touch screen. Often the determination and locking occurs at setdown, i.e., when the finger is positioned on the touch screen.

FIG. 12 is a diagram of a zoom gesture method 350, in accordance withone embodiment of the invention. The zoom gesture can be performed on amultipoint touch screen such as the multi-touch panel 24 shown in FIG.2. The zoom gesture method 350 generally begins at block 352 where thepresence of at least a first finger and a second finger are detected ona touch sensitive surface at the same time. The presence of at least twofingers can be configured to indicate that the touch is a gestural touchrather than a tracking touch based on one finger. In some cases, thepresence of only two fingers indicates that the touch is a gesturaltouch. In other cases, any number of more than two fingers indicatesthat the touch is a gestural touch. In fact, the gestural touch can beconfigured to operate whether two, three, four or more fingers aretouching, and even if the numbers change during the gesture, i.e., onlyneed a minimum of two fingers at any time during the gesture.

Following block 352, the zoom gesture method 350 proceeds to block 354where the distance between at least the two fingers is compared. Thedistance may be from finger to finger or from each finger to some otherreference point as for example the centroid. If the distance between thetwo fingers increases (spread apart), a zoom-in signal can be generatedas shown in block 356. If the distance between two fingers decreases(close together), a zoom-out signal can be generated as shown in block358. In most cases, the set down of the fingers will associate or lockthe fingers to a particular image object being displayed. For example,the touch sensitive surface can be a touch screen, and the image objectcan be displayed on the touch screen. This typically occurs when atleast one of the fingers is positioned over the image object. As aresult, when the fingers are moved apart, the zoom-in signal can be usedto increase the size of the embedded features in the image object andwhen the fingers are pinched together, the zoom-out signal can be usedto decrease the size of embedded features in the object. The zoomingtypically occurs within a predefined boundary such as the periphery ofthe display, the periphery of a window, the edge of the image object,and/or the like. The embedded features can be formed on a plurality oflayers, each of which represents a different level of zoom.

In most cases, the amount of zooming varies according to the distancebetween the two objects. Furthermore, the zooming typically can occursubstantially simultaneously with the motion of the objects. Forinstance, as the fingers spread apart or close together, the objectzooms in or zooms out at the same time. Although this methodology isdirected at zooming, it should be noted that it may also be used forenlarging or reducing. The zoom gesture method 350 may be particularlyuseful in graphical programs such as publishing, photo, and drawingprograms. Moreover, zooming may be used to control a peripheral devicesuch as a camera, i.e., when the finger is spread apart, the camerazooms out, and when the fingers are closed the camera zooms in.

FIGS. 13A-13H illustrate a zooming sequence using the method describedabove. FIG. 13A illustrates a display presenting an image object 364 inthe form of a map of North America with embedded levels which can bezoomed. In some cases, as shown, the image object can be positionedinside a window that forms a boundary of the image object 364. FIG. 13Billustrates a user positioning their fingers 366 over a region of NorthAmerica 368, particularly the United States 370 and more particularlyCalifornia 372. In order to zoom in on California 372, the user startsto spread their fingers 366 apart as shown in FIG. 13C. As the fingers366 spread apart further (detected distance increases), the map zooms infurther on Northern California 374, then to a particular region ofNorthern California 374, then to the Bay area 376, then to the peninsula378 (e.g., the area between San Francisco and San Jose Area), and thento the city of San Carlos 380 located between San Francisco and San Joseas illustrated in FIGS. 13D-13H. In order to zoom out of San Carlos 380and back to North America 368, the fingers 366 are closed back togetherfollowing the sequence described above, but in reverse.

FIG. 14 is a diagram of a pan method 400, in accordance with oneembodiment of the invention. The pan gesture may be performed on amultipoint touch screen. The pan method 400 generally begins at block402 where the presence of at least a first object and a second objectcan be detected on a touch sensitive surface at the same time. Thepresence of at least two fingers can be configured to indicate that thetouch is a gestural touch rather than a tracking touch based on onefinger. In some cases, the presence of only two fingers indicates thatthe touch is a gestural touch. In other cases, any number of more thantwo fingers indicates that the touch is a gestural touch. In fact, thegestural touch can be configured to operate whether two, three, four ormore fingers are touching, and even if the numbers change during thegesture, i.e., need a minimum of only two fingers.

Following block 402, the pan method 400 proceeds to block 404 where theposition of the two objects when the objects are moved together acrossthe touch screen is monitored. Following block 404, the pan method 400proceeds to block 406 were a pan signal can be generated when theposition of the two objects changes relative to an initial position. Inmost cases, the set down of the fingers will associate or lock thefingers to a particular image object displayed on the touch screen.Typically, when at least one of the fingers is positioned over theposition on the image object. As a result, when the fingers are movedtogether across the touch screen, the pan signal can be used totranslate the image in the direction of the fingers. In most cases, theamount of panning varies according to the distance the two objects move.Furthermore, the panning typically can occur substantiallysimultaneously with the motion of the objects. For instance, as thefingers move, the object moves with the fingers at the same time.

FIGS. 15A-15D illustrate a panning sequence based on the pan method 400described above. Using the map of FIG. 13A, FIG. 15A illustrates a userpositioning their fingers 366 over the map. Upon set down, the fingers366 are locked to the map. As shown in FIG. 15B, when the fingers 366are moved vertically up, the entire map 364 can be moved up therebycausing previously seen portions of map 364 to be placed outside theviewing area and unseen portions of the map 364 to be placed inside theviewing area. As shown in FIG. 15C, when the fingers 366 are movedhorizontally sideways, the entire map 364 can be moved sideways therebycausing previously seen portions of map 364 to be placed outside thevowing area and unseen portions of the map to be placed inside theviewing area. As shown in FIG. 15D, when the fingers 366 are moveddiagonally, the entire map 364 can be moved diagonally thereby causingpreviously seen portions of map 364 to be placed outside the viewingarea and unseen portions of the map to be placed inside the viewingarea. As should be appreciated, the motion of the map 364 follows themotion of the fingers 366. This process is similar to sliding a piece ofpaper along a table. The pressure the fingers exert on the paper locksthe paper to the fingers and when the fingers are slid across the table,the piece of paper moves with them.

FIG. 16 is a diagram of a rotate method 450, in accordance with oneembodiment of the invention. The rotate gesture can be performed on amultipoint touch screen. The rotate method 450 generally begins at block452 where the presence of a first object and a second object aredetected at the same time. The presence of at least two fingers can beconfigured to indicate that the touch is a gestural touch rather than atracking touch based on one finger. In some cases, the presence of onlytwo fingers indicates that the touch is a gestural touch. In othercases, any number of more than two fingers indicates that the touch is agestural touch. In still some other instances, the gestural touch can beconfigured to operate whether two, three, four or more fingers aretouching, and even if the numbers change during the gesture, i.e., onlyneed a minimum of two fingers.

Following block 452, the rotate method 450 proceeds to block 454 wherethe angle of each of the finger is set. The angles can be typicallydetermined relative to a reference point. Following block 454, rotatemethod 450 proceeds to block 456 where a rotate signal can be generatedwhen the angle of at least one of the objects changes relative to thereference point. In most cases, the set down of the fingers willassociate or lock the fingers to a particular image object displayed onthe touch screen. Typically, when at least one of the fingers ispositioned over the image on the image object, the image object will beassociated with or locked to the fingers. As a result, when the fingersare rotated, the rotate signal can be used to rotate the object in thedirection of finger rotation (e.g., clockwise, counterclockwise). Inmost cases, the amount of object rotation varies according to the amountof finger rotation, i.e., if the fingers move 5 degrees then so will theobject. Furthermore, the rotation typically can occur substantiallysimultaneously with the motion of the fingers. For instance, as thefingers rotate, the object rotates with the fingers at the same time.

FIGS. 17A-17C illustrate a rotating sequence based on the methoddescribed above. Using the map of FIG. 13A, FIG. 17A illustrates a userpositioning their fingers 366 over the map 364. Upon set down, thefingers 366 are locked to the map 364. As shown in FIG. 17B, when thefingers 366 are rotated in a clockwise direction, the entire map 364 canbe rotated in the clockwise direction in accordance with the rotatingfingers 366. As shown in FIG. 17C, when the fingers 366 are rotated in acounterclockwise direction, the entire map 364 can be rotated in thecounterclockwise direction in accordance with the rotating fingers 366.

It should be noted that, although FIGS. 17A-17C show the use of a thumband an index finger to invoke the rotational gesture, the use of twofingers, such as an index finger and a middle finger, can also be usedto invoke the rotational gesture.

Furthermore, in certain specific applications, two fingers may not berequired to invoke a rotational gesture. For instance, in accordancewith a preferred embodiment and as shown in FIGS. 17D and 17E, a photothumbnail can be rotated to a desired orientation (e.g., from alandscape orientation to a portrait orientation) with a single fingergesture. Specifically, upon a detection of touch in association with aselectable photo thumbnail icon 741, and wherein the touch input isgestural in that the touch detected forms a rotational or radial arcabout a center portion of the thumbnail, then that input is interpretedto be an instruction to rotate the thumbnail in accordance with thedirection of the rotation or radial arc. In accordance with a preferredembodiment, the rotation of the thumbnail icon will also cause thecorresponding file object to change the orientation configuration. Inaccordance with another embodiment, within the application of photomanagement, a detection of a rotational gesture will also invoke a snapcommand to automatically rotate the photo thumbnail by 90 degreestowards the direction of the rotation.

FIGS. 18A and 18B illustrate another example of using gestural input viaa UI element to edit a media file, such as a photograph, in accordancewith an exemplary embodiment of the invention as previously described inFIG. 10. Specifically, as shown in FIG. 18A, within a photo editorenvironment 750 in which a photo image file (e.g., a JPEG file) 752 canbe opened for editing, a UI element 751 can be provided for editingaspects of the photo. The UI element 751 can be a level slider bar foradjusting levels of certain aspects of the photograph. In the exampleillustrated in FIG. 18A, the UI element 751 can be an interface forreceiving a touch gesture to adjust the level of brightness of thephotograph. Specifically, as the tracked finger touch moves to the lefton the bar, the level of brightness is decreased, whereas the level ofbrightness increases if the tracked touch moves to the right on the UIelement. In accordance with one embodiment, the UI element is preferablytranslucent so that images of the photograph behind the UI element canstill be seen by the user. In another embodiment, the size of the photodisplayed can be reduced on the screen to make room for a separatelydisplayed UI element, which can be placed immediately below thedisplayed photo.

FIG. 18B illustrates the ability to switch the modes of gestural inputvia the UI element 751 by selectively using a single or multipletouchdown points. Specifically, as shown in FIG. 18B, a detection of asecond touchdown point on the UI element 751 will cause the mode ofoperation from brightness level adjustment to contrast level adjustment.In this instance, the movement of both touchdown points to the left orto the right will cause the level of contrast of the photograph todecrease or increase, respectively. Detection of additional touchdownpoints (e.g., three or four fingers) can also be interpreted asinstructions for switching to other modes of operation (such as zooming,hue adjustment, gamma levels, etc.). It is noted that, although FIGS.18A and 18B illustrate brightness and contrast levels to be adjusted viathe UI element 751, a user can program or customize the UI element 751to interpret the number of touchdown points to mean other forms of modesof operation. It should also be noted that the slide bar UI element 751can take other forms, such as a virtual scroll wheel.

FIG. 18C is a flow chart illustrating an algorithm association with thespecific examples discussed above in FIGS. 18A and 18B. Specifically, asshown in FIG. 18C, the UI element 751 is outputted 760 on the screen. Ifa gesture input touch is detected 761, then further determinations762-765 can be made as to how many touchdown points are associated withthe touch. Depending on the number of touchdown points detected,corresponding modes of operation can be activated at 767-769. Once theappropriate mode of operation is activated, tracking of the touchdownpoint(s) is/are detected 770 to effect 771 the corresponding adjustmentin accordance with the mode of operation. It should be noted that themodes of operation can switch at any point in time during the editingprocess in that, if the number of touchdown point(s) is/are detected 772as changed, then the process loops back to determinations 762-764 inorder to activate the new mode of operation.

FIGS. 18D and 18E illustrate using the same UI element 751 discussedabove to invoke additional actions by inputting other gesturalinstructions. Specifically, while adjusting the brightness level of thephoto displayed, a second finger can be used to effect a zoom in or zoomout action. The zoom in and zoom out action can be invoked by detectinga second touchdown point and a change in distance proximity between thetwo touchdown points. The distance change between the two touchdownpoints can be translated into a zoom in or zoom out action in accordancewith the method shown in FIG. 12 and discussed above. It should be notedthat, in accordance with one embodiment, the zoom action would not beinvoked if the second touchdown point detected remains at a constantdistance with the first touchdown point; in such a case the gesturewould be interpreted as an input for activating the second mode ofoperation (e.g., changing from brightness level adjustment to contrastlevel adjustment, as shown in FIGS. 18A and 18B).

FIGS. 19A and 19B illustrate an example of using gestural input toscroll through media files, such as photo files displayed in a photoeditor. Specifically, as shown in FIGS. 19A and 19B, a touch detectionzone 754 may be dedicated to scrolling action whereby a gesture of an upand down movement of a finger on the displayed photo 752 of the touchscreen 750 may be interpreted as a gestural input for scrolling to thenext photo 753. In accordance with a preferred embodiment, it is notnecessary to display a UI element to invoke the scrolling mode ofoperation; rather, a detection of a downward sliding action by a fingerwithin the touch detection zone 754 (e.g., a detection of downwardtracking movement of a touchdown point) may be sufficient toautomatically invoke the scrolling action. In accordance with analternative embodiment, a UI element can be displayed on the screen as avirtual vertical slide bar to indicate to the user that a scrollingaction has been activated, and the area of the touch detection zone 754for continuing the scrolling action.

In accordance with a preferred embodiment, if the downward trackingmovement detected is of more than one touchdown point (e.g., a twofinger sliding gesture), then the scrolling can be performed at 2Xspeed, similar in the fashion described above with respect to invokingscrolling action within a scrollable area.

FIGS. 19C and 19D show another form of UI element, a virtual scrollingwheel 755, for receiving gestural input to scroll the display of thephotos. In this embodiment, the virtual scroll wheel can be invoked by asimple gesture of performing a circular touch on the photo with onefinger, or a touch down of three fingers. Once the virtual scroll wheelUI element 755 can be presented, the user can “rotate” the virtualscroll wheel to scroll through the photos. In this particularembodiment, the speed of the scrolling is not controlled by how manytouchdown points are detected on the scroll wheel 755, but rather by thespeed at which the touchdown point rotates about the center of thevirtual scroll wheel 755.

FIGS. 19E and 19F illustrate the concept of FIGS. 19A and 19B on adisplay screen 781 of a digital camera 780. In accordance with apreferred embodiment, the display screen 781 of the digital camera 780can be made of a multi-touch sensitive panel, such as the multi-touchsensitive panel 2 described in FIG. 2 above.

FIG. 19E shows an embodiment where, in a playback mode of the digitalcamera 780, a detection of a vertically downward swipe gesture input ofat least one finger in a touch detection zone 782 invokes a playbackscrolling action whereby a next photo can be displayed. In accordancewith another embodiment, a downward gestural input on any part of thedisplay 781 can automatically invoke the scrolling action.

FIG. 19F shows an alternative embodiment of FIG. 19E, where a detectionof two touches are required in order to invoke playback scrolling.Specifically, a combination of a touchdown point at touch down zone 783along with a downward sliding input at or near touchdown zone 782 caninvoke a scrolling action to display the next photo. It should be notedthat the methods described in FIGS. 19A through 19E are not form factorspecific, in that the methods can be implemented on a PC monitor, alaptop monitor, a digital camera, or any type of device having a touchscreen.

FIG. 19G illustrates additional gesture that can be inputted during theplayback of media files such as photo files in accordance with anotherembodiment. Specifically, similar to the embodiments illustrated inFIGS. 18A and 18B, by distinguishing the number of touchdown points onthe touch sensitive display (i.e., the number of fingers), the samemovement can be interpreted differently. In this instance, a verticallydownward swipe gesture by two fingers may be interpreted as a gesturefor deleting the photo file, marking the photo file (for purposes suchas compiling a photo album), or any other useful commands.

FIG. 19H illustrates detecting yet other additional gestures using otherdesignated UI zones of the touch sensitive display. In this instance, adetection of a touchdown point at another designated zone 756 may beinterpreted to mean a delete, marking, or other useful commands. Inaccordance with one embodiment, the multiple touchdown zones can bedisplayed as translucent overlays to the photo file.

It should be noted that, although FIG. 19 illustrates swiping gesturesin the vertical downward direction, it is also contemplated that swipingin a vertical upward direction, or in a horizontal direction, may alsobe designated as gesture input of the same commands.

FIG. 20 illustrates one possible algorithm for implementing the methodsshown in FIGS. 19A-19F. Specifically, at the first step, one of aplurality of photos are shown 790 on a touch-sensitive display. If atouch on the display screen is detected 791, then a determination can bemade 792 as to whether the touch was a gestural input, and 793 the typeof gestural input was received (e.g., a downward tracked sliding action,a circular tracked rotation action, etc.). In accordance with thegestural input detected, a UI element (e.g., a slide bar or a virtualscroll wheel) can be outputted 794 as necessary, after which actioncorresponding to the usage of the UI element or the gestural input canbe invoked 795.

It should be noted that the methods described in FIGS. 18-20 can also beimplemented within a video environment. Specifically, during theplayback of a video file, a UI element such as a horizontal slide barshown in FIG. 18A can also be invoked and displayed, whereby, dependingon the number of touchdown points detected, a mode of operation can beactivated for changing certain adjustable aspects of the video, such asbrightness, contrast, etc. At the same time, the scrolling and zoomingmethods shown in FIGS. 19A-19F can also be effected in a similar manner,although instead of scrolling, it would be the rewind and fast forwardactions that would be performed.

Additional editing/playback functions of video files can be implementedusing gestural inputs over certain pre-existing control elements. Inaccordance with a preferred embodiment, a non-linear time playback of avideo file can be effected by selectively contracting or expanding theplayback timeline indicating bar. Specifically, FIG. 21A shows a videoapplication 790 (such as a video playback application) displays videoplayback 791 along with a progress bar 792, on which a playback queue793 indicates the time progress of the video playback.

According to a preferred embodiment, the playback queue 793 can be movedforward or backwards on the progress bar 792 to effect fast forward andrewind of the video. The queue can also be held at the same place orotherwise modulated in a non-linear speed to effect variable speedplayback or pause of the video. According to a preferred embodiment, thevideo application 790 can be displayed on a touch sensitive display, andthe position of the playback queue 793 can be manipulated via manualtouch of the queue by a finger of hand 501 at a location where the queuecan be displayed on the screen. That is, the playback queue 793 canserve both as a progress indicator as well as a UI element forcontrolling the speed and temporal location of the video playback.

In accordance with a preferred embodiment, the entire progress bar 792can serve as a UI element whereby a user can effect non-linear playbackof the video by expanding or contracting one or more sections of theprogress bar. Specifically, as shown in FIG. 21B, the UI elementprogress bar 792 can be manipulated via a two finger zoom in or zoom outgesture (as discussed above with respect to FIG. 12). In the exampleshown in FIG. 21B, a zoom in gesture invokes a expansion of the playbacktime between the 60 minute mark and the 80 minute mark. In the exampleshown in FIG. 21B, playback speed of the video becomes non-linear inthat the playback speed of the video can be slowed during the timeperiod between the 60 and 80 minute mark. Alternatively, the playbackspeed of the video can be accelerated between the 0 and 60 minute mark,and after the 80 minute mark, whereas the playback speed is normalbetween the 60 and the 80 minute mark.

FIG. 21C shows an additional UI element 794 being displayed within thevideo application 790. In this embodiment, UI element 794 can be avirtual scroll wheel whereby a user can further control the playbackspeed of the video. In combination with the manipulation of the progressbar 792, a user can first designate a section of the video for whichplayback speed is slowed, and whereby the user can use the scroll wheel794 to further modulate the playback queue 793 to control the playbackdirection and/or speed of the video.

FIG. 21D shows other additional touch sensitive UI elements that can beadded to the video application 790 for editing purposes. For instance,as shown in FIG. 21D, slide bar UI element 796 can be added to detectgestural inputs for invoking level adjustments, such as pan adjustmentor brightness, contrast, hue, gamma, etc. types of adjustments. Similarto the UI element 751 as discussed with references to FIGS. 18A-18E,slide bar UI element 796 can be used to invoke different modes ofoperation by varying the number of touchdown points on the slide bar UIelement 796.

UI element 795 can also be displayed within the video application 790 toeffect sound editing of the video. Specifically, UI element 795 caninclude a plurality of level adjustments for recording or playback ofdifferent channels or sounds or music to be mixed with the video.

In accordance with a preferred embodiment, a user of video application790 can customize which UI elements to display, and can additionallyprogram the UI elements to performed a desired function.

FIGS. 22A and 22B illustrate an example algorithm 800 for effecting themethod described with respect to FIGS. 21A-21D. Specifically, as shownin FIG. 22A, video application 790 can be launched to provide videoplayback and/or editing 802. A progress bar 792 can be displayed 803. Ifa touch is detected 804 over the progress bar 792, then a determination805 can be made as to whether the touch is a zoom in or zoom outcommand. If the touch is not detected to be a zoom in or a zoom outcommand, then the playback queue can be manipulated in accordance with atracked touch input. If the touch is detected to be a zoom gesture, thenthe portion of the progress bar at which touch is detected can bemanipulated to expand or contract according to the gestural input.

At FIG. 22B, steps 808-810 can be performed to optionally displayadditional UI elements such as the scroll wheel, sound mixer, and theslide bar level adjustment, respectively. Touch(es) can be detected atsteps 811-813, after which the appropriate functions 814-818 can beinvoked.

FIG. 23 illustrates another embodiment of the invention for manipulatingthe replay and recording of audio or musical files. As shown in FIG. 23,a music application 830 can display a pair of virtual turntables 842 and843, on which two musical records 834 and 835 are playing, the recordsbeing one of a single or a LP record. The records 834 and 835 can begraphical representations of a digital musical file (e.g., song A andsong B) that are being replayed via the music application 830. In otherwords, the records can be graphical imprints of the musical files as ifthe musical files were imprinted on physical records.

Like a pair of physical turntables, stylus 844 and stylus 855 can begraphical icon indications of a playback queue, the position of whichcan be varied by touching the queue on a touch sensitive display screenand dragging the icon to the desired position on the graphical record.The moving of the stylus would cause a jump in the playback point of thecorresponding song, as on a physical turntable.

Also like a pair of physical turn tables start/stop buttons 838 and 839can be touched by one or more fingers to toggle the start or stop/pauseof the song reproduction. Speed variants bars 840 and 841 can belinearly adjusted to control the playback speed of the songs. Windows831 and 833 can graphically reproduce the frequency representation ofthe reproduced songs, while window 832 can display the frequencyrepresentation of the actual output of the music application 832, whichcan be simply one of the songs being reproduced, or a mixed/combinationof the songs. Mixing/pan bar 850 can be manipulated to modulate ordemodulate the two songs being reproduced.

During song reproduction, the records 834 and 835 can be manipulatedsimilar to a physical record. For instance, rapid back and forthmovement of a record can cause the sound effect of a record“scratching,” as disc jockeys often do on physical turn tables.

It should be noted that the methods described above can be implementedsimultaneously during the same gestural stroke. That is, selecting,tracking, zooming, rotating and panning can all be performed during agestural stroke, which can include spreading, rotating and slidingfingers. For example, upon set down with at least two fingers, thedisplayed object (map) can be associated or locked to the two fingers.In order to zoom, the user can spread or close their fingers. In orderto rotate, the user can rotate their fingers. In order to pan, the usercan slide their fingers. Each of these actions can occur simultaneouslyin a continuous motion. For example, the user can spread and close theirfingers while rotating and sliding them across the touch screen.Alternatively, the user can segment each of these motions without havingto reset the gestural stroke. For example, the user can first spreadtheir fingers, then rotate their fingers, then close their fingers, thenslide their fingers and so on.

It should also be noted that it is not necessary to always use a humanfinger to effect gestural input. Where possible, it is also sufficientto use a pointing device, such as a stylus, to effect gestural input.

Additional examples of gestural strokes that can be used as inputs foreffecting interface commands, including interactions with UI elements(e.g., a virtual scroll wheel), are shown and described in commonlyassigned co-pending application Ser. No. 10/903,964, published as U.S.patent publication no. US2006/0026521, and application Ser. No.11/038,590, published as U.S. patent publication no. US2006/0026535, theentirety or both of which are hereby incorporated by reference.

Many alterations and modifications can be made by those having ordinaryskill in the art without departing from the spirit and scope of thisinvention. Therefore, it must be understood that the illustratedembodiments have been set forth only for the purposes of example andthat they should not be taken as limiting this invention as defined bythe following claims. For instance, although many of the embodiments ofthe invention are described herein with respect to personal computingdevices, it should be understood that the invention is not limited todesktop or laptop computers, but can be generally applicable to othercomputing applications such as mobile communication devices, standalonemultimedia reproduction devices, etc.

The words used in this specification to describe this invention and itsvarious embodiments are to be understood not only in the sense of theircommonly defined meanings, but to include by special definition in thisspecification structure, material or acts beyond the scope of thecommonly defined meanings. Thus if an element can be understood in thecontext of this specification as including more than one meaning, thenits use in a claim must be understood as being generic to all possiblemeanings supported by the specification and by the word itself.

The definitions of the words or elements of the following claims are,therefore, defined in this specification to include not only thecombination of elements which are literally set forth, but allequivalent structure, material or acts for performing substantially thesame function in substantially the same way to obtain substantially thesame result. In this sense it is therefore contemplated that anequivalent substitution of two or more elements can be made for any oneof the elements in the claims below or that a single element may besubstituted for two or more elements in a claim.

Insubstantial changes from the claimed subject matter as viewed by aperson with ordinary skill in the art, now known or later devised, areexpressly contemplated as being equivalently within the scope of theclaims. Therefore, obvious substitutions now or later known to one withordinary skill in the art are defined to be within the scope of thedefined claim elements.

The claims are thus to be understood to include what is specificallyillustrated and described above, what can be conceptionally equivalent,and what can be obviously substituted. For instance, the term “computer”or “computer system” as recited in the claims shall be inclusive of atleast a desktop computer, a laptop computer, or any mobile computingdevice such as a mobile communication device (e.g., a cellular orWi-Fi/Skype phone, e-mail communication devices, personal digitalassistant devices), and multimedia reproduction devices (e.g., iPod, MP3players, or any digital graphics/photo reproducing devices).

1. A computer implemented method for processing touch inputs using atouch sensitive display device, said method comprising the steps of:detecting a touch input at the surface of the display device, said touchinput including at least one touchdown point on the surface of thedisplay device; associating the at least one touchdown point with atleast one selectable file object displayed on the display device;identifying at least one gesture associated with the touch inputdetected; and performing at least one operation on the associatedselectable file object, wherein the at least one gesture associated withthe detected touch input is identified in part by the number ofsequential touches associated with the at least one touchdown pointdetected on the surface of the display device.
 2. The method of claim 1,wherein, if the gesture identified is a single touch by said at leastone touchdown point on the selectable file object, then the at least oneoperation performed is file select.
 3. The method of claim 1, wherein,if the gesture identified is a single touch by said at least onetouchdown point on the selectable file object followed by a dragging ofthe touchdown point on the surface of the display device, then the atleast one operation performed includes file select and drag, said dragcorrelate with a travel of the at least one touchdown point on thedisplay device.
 4. The method of claim 1, wherein, if the gestureidentified is a double sequential touch by said at least one touchdownpoint on the selectable file object, then the at least one operationincludes associating the selectable file object with a softwareapplication, launching the software operation, and activating theselectable file object with the software application.
 5. The method ofclaim 1, wherein, if the gesture identified is a single touch by said atleast one touchdown point on the selectable file object, followed by atleast one touch of another one of said at least one touchdown point onthe surface of the display device, then the at least one operationincludes displaying on the display device a plurality of selectableoperations that can be performed in association with the selectable fileobject.
 6. A computer system comprising: a touch sensitive displaydevice; means for detecting a touch input at the surface of the displaydevice, said touch input including at least one touchdown point on thesurface of the display device; means for associating the at least onetouchdown point with at least one selectable file object displayed onthe display device; means for identifying at least one gestureassociated with the touch input detected; means for performing at leastone operation on the associated selectable file object.
 7. The computersystem of claim 6, wherein, if the gesture identified is a single touchby said at least one touchdown point on the selectable file object, thenthe at least one operation performed is file select.
 8. The computersystem of claim 6, wherein, if the gesture identified is a single touchby said at least one touchdown point on the selectable file objectfollowed by a dragging of the touchdown point on the surface of thedisplay device, then the at least one operation performed includes fileselect and drag, said drag correlate with a travel of the at least onetouchdown point on the display device.
 9. The computer system of claim6, wherein, if the gesture identified is a double touch by said at leastone touchdown point on the selectable file object, then the at least oneoperation includes associating the selectable file object with asoftware application, launching the software operation, and activatingthe selectable file object with the software application.
 10. Thecomputer system of claim 6, wherein, if the gesture identified is asingle touch by said at least one touchdown point on the selectable fileobject, followed by at least one touch of another one of said at leastone touchdown point on the surface of the display device, then the atleast one operation includes displaying on the display device aplurality of selectable operations that can be performed in associationwith the selectable file object.
 11. A program embodied in acomputer-readable medium, said program containing executableinstructions for causing a computer to perform a method or processingtouch inputs, said computer including a touch sensitive display device,said method comprising the steps of: detecting a touch input at thesurface of the display device, said touch input including at least onetouchdown point on the surface of the display device; associating the atleast one touchdown point with at least one selectable file objectdisplayed on the display device; identifying at least one gestureassociated with the touch input detected; performing at least oneoperation on the associated selectable file object.
 12. The program ofclaim 11, wherein, if the gesture identified is a single touch by saidat least one touchdown point on the selectable file object, then the atleast one operation performed is file select.
 13. The program of claim11, wherein, if the gesture identified is a single touch by said atleast one touchdown point on the selectable file object followed by adragging of the touchdown point on the surface of the display device,then the at least one operation performed includes file select and drag,said drag correlate with a travel of the at least one touchdown point onthe display device.
 14. The program of claim 11, wherein, if the gestureidentified is a double touch by said at least one touchdown point on theselectable file object, then the at least one operation includesassociating the selectable file object with a software application,launching the software operation, and activating the selectable fileobject with the software application.
 15. The program of claim 11,wherein, if the gesture identified is a single touch by said at leastone touchdown point on the selectable file object, followed by at leastone touch of another one of said at least one touchdown point on thesurface of the display device, then the at least one operation includesdisplaying on the display device a plurality of selectable operationsthat can be performed in association with the selectable file object.16. A computer implemented method for processing touch inputs using atouch sensitive display device, said method comprising the steps of:detecting a touch input at the surface of the display device, said touchinput including at least one touchdown point on the surface of thedisplay device; identifying the touch input to be a scroll gestureassociated with a scrollable area displayed on the display device, saidscroll gesture including a drag of the at least one touchdown point in adirection of the intended scroll; and scrolling the scrollable area inthe direction of the intended scroll.
 17. The method of claim 16,further comprising the steps of: periodically determining the number oftouchdown points associated with the scroll gesture; and controlling thespeed of the scrolling in accordance with the number of touchdown pointsdetermined to be associated with the scroll gesture, wherein the speedof the scrolling increases with the increased number of determinedtouchdown points.
 18. The method of claim 16, further comprising a stepof displaying a scroll action icon on the display device.
 19. A computersystem comprising: a touch sensitive display device; means for detectinga touch input at the surface of the display device, said touch inputincluding at least one touchdown point on the surface of the displaydevice; means for identifying the touch input to be a scroll gestureassociated with a scrollable area displayed on the display device, saidscroll gesture including a drag of the at least one touchdown point in adirection of the intended scroll; and means for scrolling the scrollablearea in the direction of the intended scroll.
 20. The computer system ofclaim 19, further comprising: means for periodically determining thenumber of touchdown points associated with the scroll gesture; and meansfor controlling the speed of the scrolling in accordance with the numberof touchdown points determined to be associated with the scroll gesture,wherein the speed of the scrolling increases with the increased numberof determined touchdown points.
 21. The computer system of claim 19,further comprising means for causing the display of a scroll action iconon the display device.
 22. A program embodied in a computer-readablemedium, said program containing executable instructions for causing acomputer to perform a method or processing touch inputs, said computerincluding a touch sensitive display device, said method comprising thesteps of: detecting a touch input at the surface of the display device,said touch input including at least one touchdown point on the surfaceof the display device; identifying the touch input to be a scrollgesture associated with a scrollable area displayed on the displaydevice, said scroll gesture including a drag of the at least onetouchdown point in a direction of the intended scroll; and scrolling thescrollable area in the direction of the intended scroll.
 23. The programof claim 22, wherein the method further comprises the steps of:periodically determining the number of touchdown points associated withthe scroll gesture; and controlling the speed of the scrolling inaccordance with the number of touchdown points determined to beassociated with the scroll gesture, wherein the speed of the scrollingincreases with the increased number of determined touchdown points. 24.The program of claim 22, wherein the method further comprises a step ofdisplaying a scroll action icon on the display device.
 25. A computerimplemented method for processing touch inputs using a touch sensitivedisplay device, said method comprising the steps of: displaying on thedisplay device a plurality of thumbnail icons, each of said thumbnailicons representing a graphical file; detecting a touch input associatedwith at least one of said plurality of thumbnail icons; determining thetouch input to be a gesture for rotating the at least one thumbnailicon; and rotating the display orientation of the at least one thumbnailicon.
 26. The method of claim 25, wherein the display orientation of theat least one thumbnail icon is rotated by 90 degrees.
 27. The method ofclaim 25, further comprising a step of adjusting the display orientationof the graphical file associated with the at least one thumbnail icon.28. The method of claim 25, further comprising the steps of: detecting aproximity hover associated with the at least one thumbnail icon; andtemporarily enlarging the display size of the at least one thumbnailicon.
 29. A computer system comprising: a touch sensitive displaydevice; means for displaying on the display device a plurality ofthumbnail icons, each of said thumbnail icons representing a graphicalfile; means for detecting a touch input associated with at least one ofsaid plurality of thumbnail icons; means for determining the touch inputto be a gesture for rotating the at least one thumbnail icon; and meansfor causing a rotation of the display orientation of the at least onethumbnail icon.
 30. The computer system of claim 29, wherein the displayorientation of the at least one thumbnail icon is rotated by 90 degrees.31. The computer system of claim 29, further comprising means foradjusting the display orientation of the graphical file associated withthe at least one thumbnail icon.
 32. The computer system of claim 29,further comprising: means for detecting a proximity hover associatedwith the at least one thumbnail icon; and means for temporarilyenlarging the display size of the at least one thumbnail icon.
 33. Aprogram embodied in a computer-readable medium, said program containingexecutable instructions for causing a computer to perform a method orprocessing touch inputs, said computer including a touch sensitivedisplay device, said method comprising the steps of: displaying on thedisplay device a plurality of thumbnail icons, each of said thumbnailicons representing a graphical file; detecting a touch input associatedwith at least one of said plurality of thumbnail icons; determining thetouch input to be a gesture for rotating the at least one thumbnailicon; and rotating the display orientation of the at least one thumbnailicon.
 34. The program of claim 33, wherein the display orientation ofthe at least one thumbnail icon is rotated by 90 degrees.
 35. Theprogram of claim 33, further comprising a step of adjusting the displayorientation of the graphical file associated with the at least onethumbnail icon.
 36. The program of claim 33, further comprising thesteps of: detecting a proximity hover associated with the at least onethumbnail icon; and temporarily enlarging the display size of the atleast one thumbnail icon.
 37. A computer implemented method forprocessing touch inputs using a touch sensitive input device and adisplay device, said method comprising the steps of: displaying on thedisplay device a media file; displaying on the display device agraphical user interface element; detecting a touch input on the touchsensitive input device; associating the touch input with the graphicaluser interface element; determining the touch input to be a leveladjustment gesture; and adjusting the level of a graphicalcharacteristic of the displayed media file in accordance with the leveladjustment gesture.
 38. The method of claim 37, wherein the graphicaluser interface element is one of a level adjustment bar and a scrollwheel.
 39. The method of claim 37, further comprising the steps of:determining the number of touchdown points on the graphical userinterface element; and selecting, from a plurality of graphicalcharacteristics, the graphical characteristic of the displayed mediafile to be adjusted.
 40. The method of claim 37, wherein the media tileis one of a photo file, a graphical file, and a video file.
 41. Themethod of claim 39, wherein the plurality of graphical characteristicsis one of a brightness, contrast, gamma, and sharpness.
 42. The methodof claim 37, wherein the touch sensitive device is overlaid with thedisplay device.
 43. The method of claim 37, wherein the level adjustmentgesture is one of a substantially straight line movement or asubstantially circular movement by one or more touch down points on thegraphical user interface element.
 44. A computer system comprising: atouch sensitive input device; a display device; means for causing thedisplay of a media file on the display device; means for causing thedisplay of a graphical user interface element on the display device;means for detecting a touch input on the touch sensitive input device;means for associating the touch input with the graphical user interfaceelement; means for determining the touch input to be a level adjustmentgesture; and means for linearly adjusting the level of a graphicalcharacteristic of the displayed media file in accordance with the leveladjustment gesture.
 45. The computer system of claim 44, wherein thegraphical user interface element is one of a level adjustment bar and ascroll wheel.
 46. The computer system of claim 44, further comprising:means for determining the number of touchdown points on the graphicaluser interface element; and means for selecting, from a plurality ofgraphical characteristics, the graphical characteristic of the displayedmedia file to be adjusted.
 47. The computer system of claim 44, whereinthe media file is one of a photo file, a graphical file, and a videofile.
 48. The computer system of claim 46, wherein the plurality ofgraphical characteristics is one of a brightness, contrast, gamma, andsharpness.
 49. The computer system of claim 44, wherein the touchsensitive device is overlaid with the display device.
 50. The computersystem of claim 44, wherein the level adjustment gesture is one of asubstantially straight line movement or a substantially circularmovement by one or more touch down points on the graphical userinterface element.
 51. A program embodied in a computer-readable medium,said program containing executable instructions for causing a computerto perform a method or processing touch inputs, said computer includinga touch sensitive input device and a display device, said methodcomprising the steps of: displaying on the display device a media file;displaying on the display device a graphical user interface element;detecting a touch input on the touch sensitive input device; associatingthe touch input with the graphical user interface element; determiningthe touch input to be a level adjustment gesture; and linearly adjustingthe level of a graphical characteristic of the displayed media file inaccordance with the level adjustment gesture.
 52. The program of claim51, wherein the graphical user interface element is one of a leveladjustment bar and a scroll wheel.
 53. The program of claim 51, furthercomprising the steps of: determining the number of touchdown points onthe graphical user interface element; and selecting, from a plurality ofgraphical characteristics; the graphical characteristic of the displayedmedia file to be adjusted.
 54. The program of claim 51, wherein themedia file is one of a photo file, a graphical file, and a video file.55. The program of claim 53, wherein the plurality of graphicalcharacteristics is one of a brightness, contrast, gamma, and sharpness.56. The program of claim 51, wherein the touch sensitive device isoverlaid with the display device.
 57. The program of claim 51, whereinthe level adjustment gesture is one of a substantially straight linemovement or a substantially circular movement by one or more touch downpoints on the graphical user interface element.
 58. A computerimplemented method for processing touch inputs using a touch sensitiveinput device and a display device, said method comprising the steps of:displaying on the display device a media file; detecting a touch inputon the touch sensitive input device; determining the touch input to be amedia scrolling gesture; and replacing the display of the media file onthe display with a display of a second media file.
 59. The method ofclaim 58, wherein the media scrolling gesture is comprised of avertically downward swipe by one touchdown point.
 60. The method ofclaim 58, further comprising the steps of: displaying a graphical userinterface element; and associating the touch input with the graphicaluser interface element.
 61. The method of claim 60, wherein thegraphical user interface is a straight bar.
 62. The method of claim 60,wherein the graphical user interface is a scroll wheel, and whereinmedia scrolling gesture is comprised of a circular movement by at leastone touchdown point associated with the scroll wheel.
 63. The method ofclaim 58, wherein the touch input is detected at a pre-designated areaon the touch sensitive input device.
 64. The method of claim 58, furthercomprising the steps of: detecting a second touch input; determining thesecond touch input to be one of a mark or a delete gesture; and markingor deleting the media file in accordance with the detected second touchinput.
 65. The method of claim 58, wherein the touch sensitive inputdevice overlays with the display device.
 66. A computer systemcomprising: a touch sensitive input device; a display device; means forcausing the display of a media file on the display device; means fordetecting a touch input on the touch sensitive input device; means fordetermining the touch input to be a media scrolling gesture; and meansfor causing the display of a second media file, wherein the display ofthe second media file replaces the display of the media file.
 67. Thecomputer system of claim 65, wherein the media scrolling gesture iscomprised of a vertically downward swipe by one touchdown point.
 68. Thecomputer system of claim 65, further comprising: means for causing thedisplay of a graphical user interface element; and means for associatingthe touch input with the graphical user interface element.
 69. Thecomputer system of claim 65, wherein the graphical user interface is astraight bar.
 70. The computer system of claim 65, wherein the graphicaluser interface is a scroll wheel, and wherein media scrolling gesture iscomprised of a circular movement by at least one touchdown pointassociated with the scroll wheel.
 71. The computer system of claim 65,wherein the touch input is detected at a pre-designated area on thetouch sensitive input device.
 72. The computer system of claim 65,further comprising: means for detecting a second touch input on thetouch sensitive input device; means for determining the second touchinput to be one of a mark or a delete gesture; and means for marking ordeleting the media file in accordance with the detected second touchinput.
 73. The computer system of claim 65, wherein the touch sensitiveinput device overlays with the display device.
 74. A program embodied ina computer-readable medium, said program containing executableinstructions for causing a computer to perform a method or processingtouch inputs, said computer including a touch sensitive input device anda display device, said method comprising the steps of: displaying on thedisplay device a media file; detecting a touch input on the touchsensitive input device; determining the touch input to be a mediascrolling gesture; and replacing the display of the media file on thedisplay with a display of a second media file.
 75. The program of claim74, wherein the media scrolling gesture is comprised of a verticallydownward swipe by one touchdown point.
 76. The program of claim 74,further comprising the steps of: displaying a graphical user interfaceelement; and associating the touch input with the graphical userinterface element.
 77. The program of claim 74, wherein the graphicaluser interface is a straight bar.
 78. The program of claim 74, whereinthe graphical user interface is a scroll wheel, and wherein mediascrolling gesture is comprised of a circular movement by at least onetouchdown point associated with the scroll wheel.
 79. The program ofclaim 74, wherein the touch input is detected at a pre-designated areaon the touch sensitive input device.
 80. The program of claim 74,further comprising the steps of: detecting a second touch input;determining the second touch input to be one of a mark or a deletegesture; and marking or deleting the media file in accordance with thedetected second touch input.
 81. The program of claim 74, wherein thetouch sensitive input device overlays with the display device.
 82. Acomputer implemented method for processing touch inputs using a touchsensitive display device, said method comprising the steps of:reproducing a video file on the display device; displaying a playprogress bar on the display device, said play progress bar including atimeline indicating a time span of the video file; displaying a playprogress queue, said play progress queue is displayed over a location onthe play progress bar for indicating a current timing at which the videoreproduction is progressing; detecting a touch input, said touch inputbeing a time shift gesture, wherein the time shift gesture is a singletouchdown point near the display location of the play progress queue,followed by a drag action of the play progress queue along the playprogress bar; and adjusting the current timing of the video reproductionin accordance with the time shift gesture.
 83. A computer systemcomprising: a touch sensitive display device; means for reproducing avideo file on the display device; means for causing the display of aplay progress bar on the display device, said play progress barincluding a timeline indicating a time span of the video file; means forcausing the display of a play progress queue, said play progress queueis displayed over a location on the play progress bar for indicating acurrent timing at which the video reproduction is progressing; means fordetecting a touch input, said touch input being a time shift gesture,wherein the time shift gesture is a single touchdown point near thedisplay location of the play progress queue, followed by a drag actionof the play progress queue along the play progress bar; and means foradjusting the current timing of the video reproduction in accordancewith the time shift gesture.
 84. A program embodied in acomputer-readable medium, said program containing executableinstructions for causing a computer to perform a method or processingtouch inputs, said computer including a touch sensitive display device,said method comprising the steps of: reproducing a video file on thedisplay device; displaying a play progress bar on the display device,said play progress bar including a timeline indicating a time span ofthe video file; displaying a play progress queue, said play progressqueue is displayed over a location on the play progress bar forindicating a current timing at which the video reproduction isprogressing; detecting a touch input, said touch input being a timeshift gesture, wherein the time shift gesture is a single touchdownpoint near the display location of the play progress queue, followed bya drag action of the play progress queue along the play progress bar;and adjusting the current timing of the video reproduction in accordancewith the time shift gesture.
 85. A computer implemented method forprocessing touch inputs using a touch sensitive display device, saidmethod comprising the steps of: reproducing a video file on the displaydevice; displaying a play progress bar on the display device, said playprogress bar including a linear timeline indicating a linear time spanof the video file being reproduced; detecting a touch input on adesignated portion of the play progress bar, said touch input being aone of a expand or contract gesture, said expand gesture being twotouchdown points moving linearly away each other on the play progressbar, and said contract gesture being two touchdown points movinglinearly towards each other on the play progress bar; and expanding orcontracting the designated portion of the play progress bar inaccordance with the touch input detected, wherein the end points of thedesignated portion of the play progress bar is defined by the initialpositions of the two touchdown points.
 86. The method of claim 85,further comprising the steps of: if the touch input detected is anexpand gesture, contracting a remainder portion of the play progressbar; and if the touch input detected is a contract gesture, expanding aremainder portion of the play progress bar.
 87. The method of claim 86,wherein the remainder portion of the play progress bar comprises theentire play progress bar other than the designated portion of the playprogress bar.
 88. The method of claim 85, further comprising the step ofvariably controlling the reproduction speed of the video file such thatthe reproduction speed is slower if the video reproduction occurs withina time frame corresponding to an expanded portion of the play progressbar, and faster if the video reproduction occurs within a time framecorresponding to a contracted portion of the play progress bar.
 89. Themethod of claim 85, further comprising the steps of: displaying a pan UIelement; and displaying a scroll wheel UI element;
 90. The method ofclaim 85, further comprising the step of displaying a plurality of audiochannel level adjustment UI element.
 91. A computer system comprising: atouch sensitive display device; means for reproducing a video file onthe display device; means for causing a display of a play progress baron the display device, said play progress bar including a lineartimeline indicating a linear time span of the video file beingreproduced; means for detecting a touch input on a designated portion ofthe play progress bar, said touch input being a one of a expand orcontract gesture, said expand gesture being two touchdown points movinglinearly away each other on the play progress bar, and said contractgesture being two touchdown points moving linearly towards each other onthe play progress bar; and means for expanding or contracting thedesignated portion of the play progress bar in accordance with the touchinput detected, wherein the end points of the designated portion of theplay progress bar is defined by the initial positions of the twotouchdown points.
 92. The computer system of claim 91, furthercomprising means for expanding or contracting a remainder portion of theplay progress bar.
 93. The computer system of claim 92, wherein theremainder portion of the play progress bar comprises the entire playprogress bar other than the designated portion of the play progress bar.94. The computer system of claim 91, further comprising means forvariably controlling the reproduction speed of the video file such thatthe reproduction speed is slower if the video reproduction occurs withina time frame corresponding to an expanded portion of the play progressbar, and faster if the video reproduction occurs within a time framecorresponding to a contracted portion of the play progress bar.
 95. Thecomputer system of claim 91, further comprising: means for causing thedisplay of a pan UI element; and means for causing the display of ascroll wheel UI element;
 96. The computer system of claim 91, furthercomprising the step of causing the display of a plurality of audiochannel level adjustment UI element.
 97. A program embodied in acomputer-readable medium, said program containing executableinstructions for causing a computer to perform a method or processingtouch inputs, said computer including a touch sensitive display device,said method comprising the steps of: reproducing a video file on thedisplay device; displaying a play progress bar on the display device,said play progress bar including a linear timeline indicating a lineartime span of the video file being reproduced; detecting a touch input ona designated portion of the play progress bar, said touch input being aone of a expand or contract gesture, said expand gesture being twotouchdown points moving linearly away each other on the play progressbar, and said contract gesture being two touchdown points movinglinearly towards each other on the play progress bar; and expanding orcontracting the designated portion of the play progress bar inaccordance with the touch input detected, wherein the end points of thedesignated portion of the play progress bar is defined by the initialpositions of the two touchdown points.
 98. The program of claim 97,wherein said method further comprises the steps of: if the touch inputdetected is an expand gesture, contracting a remainder portion of theplay progress bar; and if the touch input detected is a contractgesture, expanding a remainder portion of the play progress bar.
 99. Theprogram of claim 98, wherein the remainder portion of the play progressbar comprises the entire play progress bar other than the designatedportion of the play progress bar.
 100. The program of claim 97, whereinsaid method further comprises the step of variably controlling thereproduction speed of the video file such that the reproduction speed isslower if the video reproduction occurs within a time framecorresponding to an expanded portion of the play progress bar, andfaster if the video reproduction occurs within a time framecorresponding to a contracted portion of the play progress bar.
 101. Theprogram of claim 97, further comprising the steps of: displaying a panUI element; and displaying a scroll wheel UI element;
 102. The programof claim 97, further comprising the step of displaying a plurality ofaudio channel level adjustment UI element.
 103. A computer implementedmethod for processing touch inputs using a touch sensitive displaydevice, said method comprising the steps of: reproducing a musical file;displaying a musical record graphical icon of the musical file data,said musical record graphical icon representing the musical file data;and graphically rotating the musical record graphical icon during thereproduction of the musical file.
 104. The method of claim 103, furthercomprising the steps of: detecting a touch input, said touch input beinga record movement gesture, said record movement gesture being arotational gesture of the musical record graphical icon by at least onetouchdown point; rotating the musical record graphical icon incorrespondence to the rotational gesture; and variably controlling thereproduction of the musical file in accordance to the rotationalmovement of the musical record graphical icon.
 105. The method of claim103, further comprising the step of displaying a stylus graphical icon,said stylus graphical icon representing a playback queue of the musicalfile.
 106. The method of claim 105, further comprising the steps of:detecting a touch input, said touch input being a stylus movementgesture, said stylus movement gesture being a touchdown point on thestylus graphical icon and dragging the stylus graphical icon from onelocation to another location on the musical record graphical icon, saidother location representing a different timing of reproduction of themusical file; and reproducing the musical file at the different timingof reproduction of the musical file.
 107. The method of claim 105,further comprising the steps of: reproducing a second musical file;displaying a second musical record graphical icon of the musical filedata, said musical record graphical icon representing the second musicalfile data; displaying a second stylus graphical icon, said second stylusgraphical icon representing a playback queue of the second musical file;and mixing the musical file data and the second musical file data toproduce a mixed musical file data.
 108. The computer system of claims 6,19, 29, 44, 66, 83, or 91 wherein said computer system is one of amobile telephone and a digital audio player.